Machine-implemented facial health and beauty assistant

ABSTRACT

An image is accepted by one or more processing circuits from a user depicting the user&#39;s facial skin. Machine learning models stored in one or more memory circuits are applied to the image to classify facial skin characteristics. A regimen recommendation is provided to the user based on the classified facial skin characteristics.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from U.S. ProvisionalApplication No. 62/614,080 filed Jan. 5, 2018, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The health and beauty industry leverages advances in technology toimprove the consumer experience with their products and services.Certain websites, for example, now avail themselves of facialrecognition techniques that locate features (eyes, cheeks, nose, lips,chin, etc.) of the human face provided through a mobile device. Suchcomputer vision techniques fail to embrace the full capabilities ofmachine learning, particularly where customization to particularconsumers is concerned. That is, for one thing, conventional health andbeauty portals lack the mechanisms by which a regimen is recommended bymachine, as opposed to recommended by a human clinician. In anembodiment, a regimen is a systematic plan or course of action intendedto improve the health and/or beauty of a human user. In the facialhealth and beauty domain, a regimen might include cleaning the skin witha specific cleanser and applying specific creams, adhering specificdietary constraints, changing sleep habits, etc.

Conventional health and beauty portals also lack the mechanisms by whichfeatures of a user's skin can be tracked over time, such as to observethe efficacy of the recommended regimen. They lack sufficientinformation explanation and advice, are not targeted separately for menand women, and suffer from false positives in detecting certainconditions (hair misidentified as wrinkles, for example). The ability torecommend a health and/or beauty regimen through data analysis, as wellas to track individual users' progress through that regimen by dataanalysis has yet to be realized on a machine.

SUMMARY

One or more images are accepted by one or more processing circuits froma user depicting the user's facial skin. In an embodiment, machinelearning models stored in one or more memory circuits are applied to theone or more images to classify facial skin characteristics, identifysignificant objects, determine beauty trends, and the like. In anembodiment, a regimen recommendation is provided to the user based onthe classified facial skin characteristics.

In an embodiment, an apparatus is provided comprising: a processingcircuit to accept at least one image depicting facial skin of a user; acommunication circuit to convey the accepted image to machine learningmodels and to receive a regimen recommendation from the machine learningmodels; and a user interface circuit to present the regimenrecommendation to the user.

In an embodiment, the processing circuit is further configured to: alertthe user that another image depicting the user's facial skin is requiredaccording to a predefined schedule; accept the other image from a userdepicting the user's facial skin; the communication circuit beingfurther configured to convey the other image to the machine learningmodels and to receive an updated regimen recommendation from the machinelearning models; and the user interface circuit being further configuredto present the updated regimen recommendation to the user.

In an embodiment, the user interface circuit is further configured topresent images of human faces to the user; the processing circuit beingfurther configured to accept input from the user that classifies facialskin characteristics from the images provided thereto through the userinterface circuit; and the communication interface circuit being furtherconfigured to convey the user input to the machine learning models astraining data.

In an embodiment, the user interface circuit is further configured topresent a user control by which the facial skin characteristics arerated on a predetermined scale.

In an embodiment, the apparatus includes a camera communicativelycoupled to the processing circuit to provide the image from the userthereto.

In an embodiment, the camera, the processing circuit, the user interfacecircuit and the communication circuit are components of a smartphone.

In an embodiment, a method is provided comprising: accepting at leastone image depicting facial skin of a user; conveying the accepted imageto machine learning models; receiving a regimen recommendation from themachine learning models; and presenting the regimen recommendation tothe user.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic block diagram of an example system configurationby which the present general inventive concept can be embodied.

FIG. 2 is a flow diagram of a simple user interaction with an embodimentof the present general inventive concept.

FIG. 3 is a schematic block diagram of example data flow of anembodiment of the present general inventive concept.

FIG. 4 is a block diagram of crowdsourced training of machine learningmodels according to an embodiment of the present general inventiveconcept.

FIG. 5 is a diagram of an example client platform device on which thepresent general inventive concept can be embodied.

FIG. 6 is a flow diagram of example crowdsourced training of machinelearning models according to an embodiment of the present generalinventive concept.

FIG. 7 is a diagram illustrating a test operation in accordance with thecrowdsourced machine learning model training.

The present inventive concept is best described through certainembodiments thereof, which are described herein with reference to theaccompanying drawings, wherein like reference numerals refer to likefeatures throughout. It is to be understood that the term invention,when used herein, is intended to connote the inventive conceptunderlying the embodiments described below and not merely theembodiments themselves. It is to be understood further that the generalinventive concept is not limited to the illustrative embodimentsdescribed below and the following descriptions should be read in suchlight.

Additionally, the word exemplary is used herein to mean, “serving as anexample, instance or illustration.” Any embodiment of construction,process, design, technique, etc., designated herein as exemplary is notnecessarily to be construed as preferred or advantageous over other suchembodiments. Particular quality or fitness of the examples indicatedherein as exemplary is neither intended nor should be inferred.

DESCRIPTION

FIG. 1 is a schematic block diagram of an exemplary facial health andbeauty assistant (FHBA) system 100 comprising an FHBA client platform110 and an FHBA service platform 120 communicatively coupled through anetwork 130. In one embodiment, FHBA client platform 110 is asmartphone, tablet computer or other mobile computing device, althoughthe present invention is not so limited. As illustrated in FIG. 1,exemplary FHBA client platform 110 comprises a processor 112, memory114, a camera 115, a user interface 116 and a communication interface118 over which an FHBA client interface 150 may be implemented. FHBAclient interface 150 provides the primary portal through which a useraccesses FHBA system 100.

In one embodiment of the present invention, FHBA service platform 120comprises one or more server computers, each comprising a processor 122,a memory 124, a user interface 126 and a communication interface. Theseresources of FHBA service platform 120 may be utilized to implement anFHBA service interface 152, machine learning logic 154 and a storagememory 156. Storage memory 156 represents a sufficient amount ofvolatile and persistent memory to embody the invention. Storage memory156 may contain vast amounts of encoded human knowledge as well as spacefor the private profile of a single user. Storage memory 156 may furtherstore processor instructions that, when executed by one or moreprocessors 122, perform some task or procedure for embodiments of theinvention. Storage memory 156 may further store user models(coefficients, weights, processor instructions, etc.) that are operablewith machine learning logic 154 to prescribe a particular regimen for auser and track the user's progress under the regimen.

Exemplary FHBA service interface 152 provides the infrastructure bywhich network access to FHBA services are both facilitated andcontrolled. FHBA client interface 150 and FHBA service interface 152communicate via a suitable communication link 145 using the signalingand data transport protocols for which communication interface 118 andcommunication interface 128 are constructed or otherwise configured.FHBA service interface 156 may implement suitable Internet hostingservices as well as authentication and other security mechanisms thatallow access only to authorized users and protect the users' privatedata. Additionally, FHBA service interface 152 may realize anapplication programming interface (API) that affords FHBA clientinterface 150 communication with, for example, machine learning logic154. Those having skill in the art will recognize other front-endservices that can be used in conjunction with the present invention.

Machine learning logic 154 provides the infrastructure for embodimentsof the invention to learn from and make predictions about data withoutbeing explicitly programmed to do so. In certain embodiments, machinelearning logic 154 implements one or more convolutional neural networks(CNNs), the models for which may be trained using open source datasetsor crowdsourced data sets, as explained below. Other machine learningtechniques may be used in conjunction with the present inventionincluding, but not limited to, decision tree learning, association rulelearning, artificial neural networks, deep learning, inductive logicprogramming, support vector machines, clustering, Bayesian networks,reinforcement learning, representation learning, similarity and metriclearning, sparse dictionary learning, genetic algorithms, rule-basedmachine learning and learning classifiers. Additional techniquesdescribed in U.S. Pat. Nos. 8,442,321, 9,015,083, 9,536,293, 9,324,022,and U.S. PG Publication No. 2014/0376819 A1, all of which areincorporated herein by reference, may be used with the presentinvention. In the descriptions that follow, it will be assumed thatmachine learning logic implements a convolutional neural network,although the present invention is not so limited. Those having skill inartificial intelligence will recognize numerous techniques that can beused in conjunction with the present invention without departing fromthe spirit and intended scope thereof.

Embodiments of the invention determine various regimens for a user basedon images of the user taken by camera 116 on FHBA client platform 110.In certain embodiments, the images of the user's face are preferablyobtained under conditions of uniform lighting that is consistent overtime. To that end and referring to FIG. 1, embodiments of the inventionprovide for a mirror device 140 that includes mirror surface 144circumscribed by a ring illuminator 142. This configuration is intendedto define a temporally constant standard of illumination. When theinvention is so embodied, temporally varying characteristics in images auser's face are more readily recognized and labeled.

FIG. 2 is a flow diagram by which an example interaction with anembodiment of the invention can be explained. The interaction of FIG. 2is simple by design and is no way intended to be limiting. Thedescription of FIG. 2 is intended to illustrate functionality of theconfiguration illustrated in FIG. 1. Further features of the invention,beyond those described with reference to FIG. 2, will be discussedbelow.

In operation 210, a user may generate an image of his face, such as bycamera 116 of FHBA client platform 110. This may be achieved with orwithout the illumination standard discussed above. In operation 215, theuser's image is sent to FHBA service platform 120. This may be achievedby suitable communication protocols shared between FHBA client platform110 and FHBA service platform 120 to realize communication link 145.

In operation 220, image analysis and machine learning is conducted toanalyze the user's skin from the images. Machine learning logic 154 mayperform analyses that determine, among other things, apparent age, i.e.,the subjective age of the user estimated from a visual appearance of theuser's face; evenness of facial skin tone (is there blotching, age/sunspots, acne scarring and other blemishes); the presence of stress asseen in under eye puffiness, dark circles, overall tone drooping ineyelids/corners of the mouth, fine lines and eye redness; hydrationlevel, often referred to as plump or slick, which presents as a lack ofashy-ness, skin flaking, dullness and fine lines; shine—a nonlinearparameter where the ideal is a moderate amount of shine; condition ofpores—a reduced appearance of pores is desirable as it provides ahealthy, youthful and smooth skin texture; the presence of acne ascharacterized by red/inflamed pimples and scarring; the presence ofwrinkles, a fold, ridge or crease in the skin may be discovered throughthe analysis; the presence of sagging, i.e., a droopy appearance of softtissue caused by elasticity reduction and the presence of crow's feet, abranching wrinkle specifically located at the outer corner of a person'seye. Other conditions of the skin may be determined by machine learninglogic 154. Further details of the analyses are provided below. Once theanalyses have been completed, as determined in operation 225, process200 may transition to operation 220, whereby the analyses results andthe prescribed regimen (products and routines) and/or updates to theregimen are sent to the user via FHBA client interface 150.

In operation 225, it is determined whether the analysis is complete and,responsive to a positive determination thereof, process 200 maytransition to operation 230, whereby FHBA service interface 152 sends arecommended regimen or updates to a regimen to FHBA client interface 150in operation 230. The user may follow the regimen as indicated inoperation 235 and, in operation 240 it is determined whether a newinterval has commenced. If so, process 200 reiterates from operation210. FHBA client interface 150 may access calendars and timers (as wellas GPS) onboard FHBA client platform 110 as well as access tonetwork-accessible calendars on network 130. Accordingly, once a week,say, FHBA client interface 150 may remind the user to take a picture ofhis face, i.e., remind him of the new interval. Over time, FHBA system100 can determine from the images taken at each interval whether therecommended regimen is working and, if not, FHBA system 100 may revisethe regimen, e.g., change a product, recommend further lifestylechanges, make a doctor's appointment, etc.

FIG. 3 is a diagram of data flow between an exemplary FHBA clientinterface 150 and services of FHBA service platform 120. It should benoted that, in FIG. 3, FHBA service interface 152 has been omitted toavoid unnecessary congestion in the figure. However, those having skillin the relevant arts will recognize the operation of an FHBA serviceinterface 152 to control and facilitate the data flow depicted in FIG.3.

As illustrated in FIG. 3, machine learning logic 154 may comprise a skinanalyzer 330, facial appearance progression generator 335 and a regimenrecommendation generator 340 and may be communicatively coupled to auser account database 310 and a product database 320. Machine learninglogic 154 may train and utilize machine learning models 370 to recommendregimens and to track the progress of the user under the regimen. Asthose skilled in machine learning will attest, training may involveselecting a set of features, e.g., apparent age, evenness, stress,hydration, shine, pores, acne, wrinkles, sagging, crow's feet, etc., andassigning labels to image data that reflects the presence or prominenceof those features. The assigning of labels may be performed by a subjectmatter expert or, as explained below, through crowdsourced data. Takingthe assigned labels as ground truth, machine learning logic 154 mayconfigure models 370 to predict the degree to which the features arepresent in a test image, which may change over time. The presentinvention is not limited to a particular model representation, which mayinclude binary models, multiclass classification models, regressionmodels, etc.

Exemplary user account database 310 contains the data of all users ofFHBA system 100 in a secure manner. This includes user profile data,current and past user photos 357 for each user, current and past skinanalyses 358 for each user, current and past product recommendations 362and current and past routine recommendations 364 for each user.

Exemplary product database 320 contains the data of different productsthat can be used in a regimen. Product database 320 may contain recordsreflecting the product names, active and inactive ingredients, labelinformation, recommended uses, and so on. In certain embodiments, asillustrated as product input 354, the user (and other users of FHBAsystem 100) may provide feedback on different products and may enterproducts not already in product database 320. The present invention isnot limited to particular products that can be entered in productdatabase 320.

Skin analyzer 330 is constructed or is otherwise configured to classifyvarious skin conditions or artifacts from imagery of a user's face usingmachine learning techniques over models 370. In certain embodiments,photographic images 352 of a user's face are provided to skin analyzer330 for analysis. Skin analyzer 330 may implement image preprocessingmechanisms that include cropping, rotating, registering and filteringinput images prior to analysis. After any such preprocessing, skinanalyzer 330 may apply models 370 to the input image 357 to locate,identify and classify characteristics of the user's facial skin.

Facial appearance progression generator 335 may operate on the user'sfacial images to portray how the user's face would appear sometime inthe future. Such progression may be in age, for which age progressiontechniques may be deployed, or may be in appearance resulting fromadherence to a regimen. A progressed image 356 may be provided to theuser through FHBA client interface 150.

Regimen recommendation generator 340 may operate on analysis results 358obtained from skin analyzer 430 towards prescribing a regimen to theuser. Models 370 may be trained to predict what products and routines(treatment, cosmetic and lifestyle recommendations, etc.) would beeffective in meeting the user's goal with regard to facial skincharacteristics identified in the skin analysis. Regimen recommendationgenerator 340 may format the analysis results 358 of skin analyzer 330as a query into, for example, product database 320 based on knowledgeencoded on models 370. In response, product database 320 may returnproduct data and metadata 366, and product recommendations 362 androutine recommendations 364 may be provided to FHBA client interface150.

As indicated above, training of models 370 may be achieved by labelingof image data by an expert. However, in lieu of an expert, certainembodiments of the invention utilize crowdsourced data as training data.FIG. 4 is a diagram of such an embodiment of the invention. Duringtraining, users 410 are presented a set of training images 420 overwhich they are asked to characterize facial skin characteristics and/orfacial features. In one embodiment, a suitable scale is constructed,(e.g., integers 1-10) with which users can rate the severity orprominence of the feature. For example, each of users 410 (over time)are presented a large number of facial images and is walked through aset of questions regarding features and/or skin characteristics of theperson in the image. Using the scale (1-10), each user 410 is asked torate the prominence of each of the features (e.g., apparent age,evenness, stress, hydration, shine, pores, acne, wrinkles, sagging,crow's feet, etc.). The answers to the questions may serve as labelsused for training machine language logic 154.

Referring to FIG. 5, there is illustrated an exemplary FHBA clientplatform 110 in the form of a smartphone having a touchscreen 510 asuser interface 118. Exemplary FHBA client interface 150 is implementedon the computational resources of FHBA client platform 110 as discussedwith reference to FIG. 1. FHBA client interface 150 may present aphotograph of a person's face in image area 520 and may present via text142 “On a scale from 1-10, where “1” means ‘invisible’ and “10” for‘prominently present,’ how would you rate the presence this person'scrow's feet?” A suitable user interface control 144 (slider controlillustrated in FIG. 5) may be implemented on FHBA client interface 150that allows a user to input its rating.

FIG. 6 is a flow diagram of a crowdsourced training process 600 withwhich the present invention may be embodied. In operation 610, atraining image may be provided to FHBA client interface 150. A set oftraining images 420 may have been preselected as including illustrativeexamples of the skin characteristics of interest. In operation 620, theuser is provided with a first question and waits for an answer (rating)in operation 630. Such question might be, for example, “On a scale of 1to 10, where ‘1’ is ‘invisible’ and ‘10’ is “highly prominent,” howwould you rate this models acne?” When the user has answered thequestion, as determined in operation 630, the user's answer may beformatted into a label suitable for machine training of machine learninglogic 154 in operation 640. In operation 650, it is determined whetherall questions relating to the currently displayed image have beenanswered. If not, process 600 may transition to back to operation 620,whereby the next question is presented. If all questions have beenanswered, as determined at operation 650, it is determined in operation560 whether all training images have been presented. If not, process 600may transition to back to operation 610, whereby the next training imageis presented. If all training images have been presented, as determinedat operation 660, the labeled images may be used to train models 370 inoperation 670.

It is to be understood that all iterations in process 600, e.g.,presenting next questions in operations 620 and 650 and/or presentingnext images in operations 610 and 660, need not be performed in any onesitting. For example, the user may be prompted to answer a singlequestion at a time (e.g., every time the user logs on) and it is onlyover time that all questions and images are presented to any one user.Alternatively, users may be selected to answer all questions for allimages in a single sitting. Over a large number of users and/or facialimages, many labels may be generated for training models 370, where thestatistical trends underlying such training reflect public views asopposed to those of a human expert.

FIG. 7 illustrates an example test operation in accordance with thecrowdsourced training discussed above. A test image 710, i.e., a user'sown image, may be presented to machine learning logic 154, whichanalyzes the image per the models trained on the crowdsourced data 720.As illustrated in the figure, machine learning logic 154 estimates that80% of people surveyed would rate the user's crow's feet a 7 out of 10in terms of prominence, as indicated at 722. Accordingly, machinelearning logic 154 may recommend a regimen, (e.g., a cream speciallyformulated for crow's feet and recommended application instructions),based on the severity score of 7.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be, for example, but is notlimited to, an electronic, magnetic, optical, electromagnetic, infrared,or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific examples (a non-exhaustivelist) of the computer readable storage medium would include thefollowing: an electrical connection having one or more wires, a portablecomputer diskette, a hard disk, a solid state disk, a random accessmemory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), an optical fiber, a portablecompact disc read-only memory (CD-ROM), an optical storage device, amagnetic storage device, a phase change memory storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, e.g., an object oriented programming languagesuch as Java, Smalltalk, C++ or the like, or a conventional proceduralprogramming language, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider). It is to be understood that the software for the computersystems of the present invention embodiments may be developed by one ofordinary skill in the computer arts based on the functional descriptionscontained in the specification and flow charts illustrated in thedrawings. Further, any references herein of software performing variousfunctions generally refer to computer systems or processors performingthose functions under software control.

The computer systems of the present invention embodiments mayalternatively be implemented by any type of hardware and/or otherprocessing circuitry. The various functions of the computer systems maybe distributed in any manner among any quantity of software modules orunits, processing or computer systems and/or circuitry, where thecomputer or processing systems may be disposed locally or remotely ofeach other and communicate via any suitable communications medium (e.g.,LAN, WAN, Intranet, Internet, hardwire, modem connection, wireless,etc.).

The foregoing examples are illustrative of certain functionality ofembodiments of the invention and are not intended to be limiting.Indeed, other functionality and other possible use cases will beapparent to the skilled artisan upon review of this disclosure.

The invention claimed is:
 1. An apparatus comprising: a processingcircuit to accept at least one image depicting facial skin of a user; acommunication circuit to convey the accepted image to machine learningmodels and to receive a regimen recommendation from the machine learningmodels; and a user interface circuit to present the regimenrecommendation to the user, wherein the processing circuit is furtherconfigured to: alert the user that another image depicting the user'sfacial skin is required according to a predefined schedule; accept theother image from a user depicting the user's facial skin; thecommunication circuit being further configured to convey the other imageto the machine learning models and to receive an updated regimenrecommendation from the machine learning models; and the user interfacecircuit being further configured to present the updated regimenrecommendation to the user.
 2. The apparatus of claim 1, wherein theuser interface circuit is further configured to present images of humanfaces to the user; the processing circuit being further configured toaccept input from the user that classifies facial skin characteristicsfrom the images provided thereto through the user interface circuit; andthe communication interface circuit being further configured to conveythe user input to the machine learning models as training data.
 3. Theapparatus of claim 2, wherein the user interface circuit is furtherconfigured to present a user control by which the facial skincharacteristics are rated on a predetermined scale.
 4. The apparatus ofclaim 1, further comprising a camera communicatively coupled to theprocessing circuit to provide the image from the user thereto.
 5. Theapparatus of claim 1, wherein the camera, the processing circuit, theuser interface circuit and the communication circuit are components of asmartphone.
 6. A method comprising: accepting at least one imagedepicting facial skin of a user; conveying the accepted image to machinelearning models; receiving a regimen recommendation from the machinelearning models; and presenting the regimen recommendation to the user,wherein the method further comprises: alerting the user that anotherimage depicting the user's facial skin is required according to apredefined schedule; accepting the other image from a user depicting theuser's facial skin; conveying the other image to the machine learningmodels; receiving an updated regimen recommendation from the machinelearning models; and presenting the updated regimen recommendation tothe user.
 7. The method of claim 6 further comprising: presenting imagesof human faces to the user; accepting input from the user thatclassifies facial skin characteristics from the images provided thereto;and conveying the user input to the machine learning models as trainingdata.
 8. The method of claim 7 further comprising: presenting a usercontrol by which the facial skin characteristics are rated on apredetermined scale.